Widen sizeInBytes beyond 32-bit Int so files over 2 GB are queryable — Closes #83 - #99

Summary

Introduce a BigInt custom scalar — a signed 64-bit integer serialized as a JSON number — and apply it to sizeInBytes on both FileMetadataType (output) and FileMetadataInput (input filter).

GraphQL's specification fixes Int at 32 bits, so any file over 2,147,483,647 bytes could not be represented: the field resolved to null and the response carried an Int cannot represent non 32-bit signed integer value entry in errors, degrading a whole page of results to a partial one. That is the common case rather than an edge case — every one of the 3,581 ENCODE .hic files runs 6–51 GB, and the larger 4DN mcools exceed the ceiling too.

The wire representation is the real decision here, so it is worth stating explicitly. GraphQL has no 64-bit Int, and the two realistic options are a BigInt that serializes as a JSON number and one that serializes as a String. This PR chooses the number.

The case against a number is real but does not bind here: a JSON number above Number.MAX_SAFE_INTEGER (2^53-1) is not exactly representable in JavaScript, and the consumer is a browser client (Gosling Designer), so "fits in 64 bits" is genuinely not the same as "the client can read it". But 2^53 bytes is roughly 9 PB. No file this API serves — or plausibly ever will — comes within five orders of magnitude of that ceiling, so the precision hazard is theoretical while the ergonomic cost of a String is immediate: every consumer of sizeInBytes would have to parse before comparing or summing, and a client that forgot would get silently wrong arithmetic ("6262125716" + 1 is "62621257161" in JavaScript) rather than a loud failure. A number keeps sizeInBytes directly usable and keeps the JSON body byte-for-byte what a client already expected, only correct.

The scalar's schema description records the 2^53 caveat so the reasoning survives in the published contract, and _coerce_big_int rejects non-integers (including true/false, since bool subclasses int in Python) and anything outside the signed 64-bit range, in both directions. The 64-bit bound is not arbitrary: it is exactly where BSON stops, so no value the scalar admits can fail on the way to MongoDB. Staying under 2^53 is deliberately not enforced — that would invent a second, softer limit inside a type whose name promises 64 bits — so it is documented as an admission criterion for routing any future field through BigInt.

What breaks for a client that assumed Int. Four things. A query declaring query Q($s: [Int!]) and passing it to sizeInBytes now fails variable-type validation and must declare [BigInt!]. Generated clients must re-run codegen against the new SDL and add a scalar mapping — graphql-codegen silently widens an unrecognised custom scalar to any, so without scalars: { BigInt: 'number' } the field loses its type with no build failure. Any client validating responses against a stored copy of the schema must refresh it. And a client sending an integral float (1234.0) is now rejected, where Int coerced it. A client that merely reads sizeInBytes out of the JSON response needs no change at all — it was already receiving a JSON number, and now receives a correct one instead of null.

The break is also not atomic on the server side, and the README now records the way around that: the leaf type only has to be named when a client declares a variable for it, so a consumer that passes the filter as an inline literal or hoists the variable to the whole [FileMetadataInput!] validates against the old schema and the new one alike — which makes a rolling deploy, the dev/prod schema skew, and a SHA rollback all non-events.

The widening lands on the input filter as well as the output field. Widening only the output would leave exactly the files it newly exposes unfilterable by size, which is half the bug.

Closes #83

Proposed changes

BigInt scalar and a scoped override mechanism

src/cfdb/api/gql/types.py defines BigInt alongside the existing ObjectIdScalar, following the same module-level @strawberry.scalar structure. One coercion function serves both serialize and parse_value, because the wire form is symmetric — the value that goes out is the value that comes back.

The output types are generated from the Pydantic models by runtime introspection, so a bare int maps to GraphQL Int. Rather than widening every int in the schema, _SCALAR_OVERRIDES maps (model, field name) to a replacement scalar and annotate() consults it first. Keying on the pair rather than the field name matters: ExtraFile.file_size is a same-named-shaped sibling on a different model that must stay Int, and so must totalCount, fileCount, and the page/pageSize arguments. _substitute_scalar preserves the Optional wrapper so Optional[int] becomes Optional[BigInt].

src/cfdb/api/gql/inputs.py is hand-written, so FileMetadataInput.size_in_bytes changes type directly.

Regenerate schema.graphql, and keep it that way

schema.graphql is a generated artifact that nothing in the repo regenerated or verified — the strawberry CLI is not an installed extra, so refreshing it meant reconstructing the print_schema incantation by hand, and nothing failed when the checked-in copy went stale. Add scripts/export_schema.py and a make schema target, plus a test asserting the file on disk matches what the live schema renders. That gap predates this change but is worth closing in the PR that first exercises it: the SDL is what clients codegen against, so a stale copy ships a wrong public contract silently.

Documentation

Add a Custom Scalars section to README.md recording the wire-representation decision and the three concrete client breakages, and note make schema in the Makefile targets table.

Test cases

Review round 1

Seven independent principal-engineer reviewers. No reviewer found a correctness, data-integrity, concurrency, or security defect; both blocking findings were Python test-guide MUST violations. Remediated: the new HTTP test now uses the mocker fixture rather than unittest.mock; the declared range is pinned by a Hypothesis round-trip property plus a rejection property, and both inclusive bounds are named cases (the six original examples were all non-negative, so _INT64_MIN was never exercised in the accepting direction); _substitute_scalar raises on any annotation it cannot express faithfully, instead of silently flattening a list into a scalar; _coerce_big_int raises GraphQLError rather than ValueError, so a malformed filter value on an unauthenticated endpoint no longer writes a stack trace at ERROR; the drift test calls the generator it names; the ExtraFileType.fileSize assertion is split out and reworded as a deferral rather than a rule; and the _JS_SAFE_INTEGER_MAX comment no longer describes a guard the code does not implement.

Four findings were rejected, with reasons recorded in the review document: migrating both scalars off the deprecated strawberry.scalar class form (would have to move ObjectIdScalar and the Schema(...) call — scope this issue does not carry; five of the seven reviewers who raised it agreed no change was needed here); widening ExtraFile.file_size to BigInt (a different model, and #83 scopes this to sizeInBytes — the test that would otherwise have defended the bug is reworded instead); renaming or relocating tests/test_metadata_endpoint.py (tests/test_cors.py is the in-repo precedent for this exact shape, and with the datastore mocked it is not a true integration test either); and a registry-walking parity test over _SCALAR_OVERRIDES (would have to import a private dict and would restate the implementation; the new import-time TypeError covers the realistic failure).

Two adjacent defects are noted and left for follow-ups: size_in_bytes is stored as a BSON string for 4DN and HuBMAP because the C2M2 TSV path never narrows it, so exact-match size filtering is inert for those DCCs regardless of the scalar's width (the README now says so); and the field is indexed on the raw file collection but not on the materialized files collection the resolvers actually query.